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■ Abstract 

(N ; 

We consider the unconstrained L2-Lp minimization: find a minimizer of HAx— &|l2+A||x||^ 

. for given A € i?™^", b € i?™ and parameters A > 0, p G [0,1). This problem has been 

^ I studied extensively in variable selection and sparse least squares fitting for high dimensional 

' . data. Theoretical results show that the minimizers of the L2-Lp problem have various 

CO ' attractive features due to the concavity and non-Lipschitzian property of the regularization 

^ function || • In this paper, we show that the Lq-Lp minimization problem is strongly 

I NP-hard for any p £ [0, 1) and q > 1, including its smoothed version. On the other hand, 

. we show that, by choosing parameters (p, A) carefully, a minimizer, global or local, will have 

• ' certain desired sparsity. We believe that these results provide new theoretical insights to 

^ . the studies and applications of the concave regularized optimization problems. 

! Keywords. Nonsmooth optimization, nonconvex optimization, variable selection, 

I sparse solution reconstruction, bridge estimator. 
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m ■ 

^ : 

5 ■ 1 Introduction 

in : 

I In this paper, we consider the following L2-Lp minimization problem: 

: Minimize^,. fp{x) := \\Ax - bg + H^Wp (1) 

^ I where data and parameter A = (ai,...,a„) G R'^^"-,0 / & G R"^, A > and < p < 1, and 



variables x G i?" . This regularized formulation has been studied extensively in variable selection 
and sparse least squares fitting for high dimensional data, see [U [21 [3l |H [5l [H [9l [101 [HI [121 [13] 
and references therein. Here, when p = 0, 

Ikllo = Ikllo = \{i ■ Xi / 0}| 
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that is, the number of nonzero entries in x. 

The original goal of the model was to find a least squares solution with fewer nonzero entries 
for an under-determined linear system that has more variables than the data measurements. 
For this purpose, people considered the regularized L2-LQ problem. For instance, the variable 
subset selection method can be viewed as the L2-L0 problem, which is the most popular method 
of regression regularization used in statistics [6]. 

However, the Lq regularized problem is difficult to deal with because of the discrete structure 
of the 0-norm, while the solvability of the L2-Lp problem for p £ (0, 1) can be derived from the 
continuity and level boundedness of fp. A (global) minimizer of the L2-Lp problem is also called 
a bridge estimator in statistical literature [6j and has various nice properties including the oracle 
property [H llOt [TT]. Moreover, theoretical results show that in distinguishing zero and nonzero 
entries of coefficients in sparse high-dimensional approximation, the bridge estimators have 
advantages over the Lasso estimators that minimize the following convex L2-L1 minimization 
problem: 

Minimize^,. := \\Ax - bW^ + A||x||i. (2) 

Due to these advantages, researchers have been interested in the Lp regularization problem 
for < p < 1. However, the L2-Lp problem ([1]) is a nonconvex, non-Lipschitz optimization 
problem. There are not many optimization theories on analyzing this type of problems. Many 
practical approaches have been developed to tackle the problem see, e.g., [H [2l [3t [TOl [T2]: 
but there is no globally convergent algorithm that guarantees to find a global minimizer or 
bridge estimator. 

To the best of our knowledge, the computational complexity of the L2-Lp minimization 
problem remains an open problem. One may attempt to draw a hardness result from the 
following problem: 

Minimize , . 

Subject to Ax = b, 

which is shown in [9] to be strongly NP-hard for p £ [0, 1); or the problem 

Minimize ll^^llo f,\ 

Subject to \\Ax - b\\2 < e, ^ ' 

which is shown in [T3] to be NP-hard for certain e. From a complexity theory perspective, an 
NP-hard optimization problem with a polynomially bounded objective function does not admit 
a polynomial-time algorithm, and a strongly NP-hard optimization problem with a polynomially 
bounded objective function does not even admit a fully-polynomial-time approximation scheme 
(FPTAS), unless P=NP [IB]. 

Indeed, the L2-Lp problem ([1]) can be viewed as a quadratic penalty problem of problem 
Intuitively, solving an unconstrained penalty optimization problem is easier than solving the 
constrained optimization problem. Unfortunately, we show that this is not true. More precisely, 
we show that finding a global minimizer of L2-Lp problem ([1]) remains strongly NP-hard for all 
< p < 1 and A > 0, including its smoothed version. We also extend the strong NP-hardness 
result to the Lq-Lp minimization problem for q > 1. 

On the positive side, we present a sufficient condition on the choice of A for the desired 
sparsity of all minimizers, global or local, of the L2-Lp problem for given (A,b,p), as long as 
their objective value is below that of the all-zero solution. Under this condition, any such a 
local optimal solution of problem ([1]) is a sparse estimator to the original problem. This may 
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explain why many methods, e.g., [U O O [TOl [12], have reported encouraging computational 
results, although what they calculate may not be a global minimizer. 

The remainder of this paper is organized as follows: in Section 2, we present sufficient 
conditions on the choice of A to meet the sparsity requirement of global or local minimizers of 
the L2-Lp minimization problem. In general, when A is sufficiently large with respect to data 
{A, b) and p, the number of nonzero entries in any minimizer of the problem must be small. In 
Section 3, we prove that the Lq-Lp minimization problem: 

Minimize^. fq^p{x):=\\Ax-b\\l + \\\x\\l (5) 

is strongly NP-hard for any given 0<p< 1, g> 1 and A > 0. We then extend our hardness 
result to its smoothed version: 

Minimize, f^^^^^{x) := \\Ax - h\\l + \YJl^^{\x^\ + ef (6) 

for any given 0<p<l,(7>l,A>0 and e > 0, even though the objective function in this case is 
Lipschitz continuous. Thus, changing the non-Lipschitz regularization model (l5|) to a Lipschitz 
continuous model ^ gains no advantage in terms of computational complexity. Finally, we 
show that our results are consistent with the existing findings from statistical literature, but 
give more specific bounds on choosing regularization parameters. We also illustrate that for the 
purpose of finding a least squares solution with a targeted number of nonzero entries, finding 
a local minimizer of problem ([1]) is likely to accomplish the same objective as finding a global 
minimizer does. 

In the rest of the paper, we define z'' = if z = and z'' = 1 if z ^ 0. We use (x • y) to 
represent the vector {xiyi, . . . ^XnVn)'^ £ and || • || to denote the L2 norm. 



2 Choosing the parameter A for sparsity 

In applications like variable selection and sparse solution reconstruction, one wants to find least 
square estimators with no more than k nonzero entries. On the other hand, one obviously 
wants to avoid the all-zero solution. The L2-Lp regularized approach is to first solve L2-Lp 
problem ([T| to find a minimizer. Then, eliminate all variables who have zero values in the 
minimizer, and solve the least square problem using only remaining variables. Thus, the key 
is to control the support size of minimizers of problem ([1]) such that it does not exceed k, and 
this is typically accomplished by selecting a suitable A. We now give a sufficient condition on A 
for the minimizers of the L2-Lp problem to have desirable sparsity. 

Theorem 1. Let 

f In \ ^/"^ 

I3{k) = F/^-^ i r , ] llfef"^, a=max\\aif, 1 < k < n. (7) 

\P[l — P) J l<i<n 

The following statements hold. 

(1) If \> Pik), any minimizer x* of L2-Lp problem ^^ satisfies ||x*||o < k for k >2. 

(2) If X > X* = is the unique minimizer of L2-Lp problem ([7]j. 

(3) Suppose that set C := {x \ Ax = b} is non-empty. Then, if X < j^Kp for some x^ G C , 
any minimizer x* of L2-Lp problem ([ip satisfies ||x*||o > 1. 
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f,{x*) = \\Ax* - b\\ + A||x*||^ > X\x*\P > A ( ) >\\bf = /(O). 



Proof. Suppose that x* 7^ is a global minimizer of the L2-Lp problem ([T]). Let B = At G 
j^mx|T|^ where T =support(x*) and \T\ = ||x*||o is the cardinality of the set T. By Theorem 2.1 
and Theorem 2.3 in [3J, the columns of B are linearly independent and x* must satisfy 

2B^{Bx*T -b)+ pX{\x*t\p-^ ■ {x*^)) = 0. (8) 

This implies Ax* — b = i?x^ — b ^ 0. Hence we have 

Uix*) = \\Ax* -bf + X\\x*r^ > X^\x*\P > X\T\ f^P^^zAY^^' , (9) 

where the last inequality is from the lower bound theory for local minimizers of ([T|) in [3l 
Theorem 2.1]. 

(1) Suppose that A > (3{k). If x* is a nonzero minimizer of ([1]) with ||x*||o > k > 1, then 
from ([9]) and the definition of I3{k) in ([7]), we have 

uix*) > ^a2/(^-) [^^^y^'~'^ > \\bf = m. 

This contradicts to that x* is a minimizer of ([TJ. Hence ||x*||o < k. 

(2) Suppose A > /3(1). If x* is a nonzero minimizer of ([1]), then there is i such that x* / 

and 

'xp{i-p)y/^^-p^ 

2a 

This contradicts to that x* is a minimizer of ([1]). Hence, x = is the unique solution of ([1]). 

(3) Note that /p(0) = \\bf and /p(xc) = A||xc||^ for Xc G C. Therefore, if 

A < for some Xc G C (10) 

then /p(0) > fp{xc)- Since Xc is not a stationary point of L2-Lp problem [3], there is x near Xc 
such that fp{xc) > fp{x). Hence x = cannot be a global minimizer of □ 

Remark 1 It was known that x = is a local minimizer of the L2-Lp problem ([T|) for any value 
of A > [3j, and x = is a global minimizer of ([1]) for a "sufficiently large" A [1^. Theorem 
[H for the first time, establishes a specific bound /3(1), such that x = is the unique global 
minimizer of ([T|) for A > /3(1). An important algorithmic implication of Theorem [1] is that, for 
given data {A, b) and p, choosing A > (3{k) for a small constant k does not help to solve the 
original sparse least squares problem. For a small constant k, say from 1 to 3, one might be 
better off to enumerate all combinations of solutions, each with no more than k nonzero entries, 
to find a minimizer. This can be done in a strongly polynomial time of the problem dimensions. 

One may be also interested in the relation of A and the support sizes of local minimizers of 
L2-Lp problem ([TJ. We present the following result for the sparsity of certain local minimizers 
of©. 

Theorem 2. Let 

7(fc)=F-^(^yii6f-^. (11) 

If X> "y{k), then any local minimizer x* of problem ([T]j, with fp{x*) < /p(0) = \\b\\'^ , satisfies 
||x*||o < k for k >2. 
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Proof. Note that ([8]) holds for any local minimizer of L2-Lp problem ([T]). By Theorem 2.3 in 

[3], for any local minimizer x* of L2-Lp problem ([T]) in the level set {x : fp{x) < /p(0)}, we 
have 

fp{x*) = \\Ax* -bf + X\\xX > > A|r| (^pp^J > (12) 
where T =support(x*). If \T\ = \\x*\\o > k >1, then 

which is a contradiction. □ 

Theorem [T] concerns global minimizers of L2-Lp problem ([T|) while Theorem [5] concerns its 
local minimizers in the level set {x : fp{x) < /p(0)}. Since a; = is a trivial local minimizer for 
problem ([T]), we believe any good method would likely find a minimizer that at least is better 
than X = 0. Below, we use an example to illustrate the bounds presented in Theorems [1] and [2j 
Example 2.1 Consider the following L2-L1/2 minimization problem 

Minimize f{x) := (xi + X2 — 1)^ + A(y^|3;i| + y^|a;2|). (13) 
From A = (1, 1), 6 = 1 and Xc = (1,0), we easily find these data in Theorem [T] and Theorem [51 

a = l, \\b\\ = l, m = 8'/'k-'/', j% = h i{k)=^2^/^k-^/\ 

For k = 2, we have /3(2) = 1. Using parts 1 and 3 of Theorem [U we can claim that any 
minimizer x* of (|13|) with A = 1 satisfies ||2;*||o = 1. Using part 2 of Theorem [H we can claim 
that x = is the unique minimizer of ()13p with A > /3(1) = 8^^^. The lower bound /3(1) can 
be improved further. In fact, we can give a number /3* < /3(1) such that re = is the unique 
minimizer of ()13p with A > /3* by using the first and second order necessary conditions for 

©• 

For A = < 8^/^, it is easy to see that {xi,X2) = (1/3,0) and {xi,X2) = (0, 1/3) are two 
vectors satisfying 

2x1 (a^i + X2 - 1) + ^ \/|xi| = 0, 2x2 (xi + X2 - 1) + ^ Vl^^l = 0, 

and 

= 2( '^^'^^ \_^( \ ^ p 

^ ^ V 2;i2;2 J 4 V J 

However, since the third order derivative of g{t) := /((1/3 + t)ei) (or g{t) := /((1/3 + t)e2)) is 
strictly positive on both side of t = 0, (xi,X2) = (1/3,0) and (xi,X2) = (0,1/3) are not local 
minimizers. Moreover, these two vectors are the only nonzero vectors satisfying both first and 
second order necessary conditions. We can claim that x = is the unique global minimizer of 
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Our theorems reinforce the findings from statistical hterature that global minimizers of 
the L2-Lp regularization problem may have many advantages over those from other convex 
regularization problems, and the new results actually give precise bounds on how to choose A 
for desirable sparsity. The remaining question: is the L2-Lp regularization problem ([T|) tractable 
for given A > and < p < 1? Or more specifically, is there an efficient or polynomial-time 
algorithm to find a global minimizer of problem ([1])? Unfortunately, we prove a strong negative 
result in the next section. 

3 The L2-Lp problem is strongly NP-hard 

As we mentioned earlier, one may attempt to draw a negative result directly from constrained 
Lp problem ([3]) or (jl]). However, it is well known that the quadratic penalty function is not exact 
because its minimizer is generally not the same as the solution of the corresponding constrained 
optimization; see, e.g., |14) . For example, the all-zero vector is a local minimizer of the L2-Lp 
problem ([1]), but it may not even be feasible for the Lp problem ([3]). On the other hand, the set 
of all basic feasible solutions of ^ is exactly the set of its local minimizer [9] , but such a local 
minimizer of ([3]) may not even be a stationary point of problem ([1]). In fact, there is no A > 
such that X, any feasible solution of problem ([3]), satisfies the first order necessary condition of 
L2-Lp problem ([T|). 

Another difference between Q and ([1]) is the following: it has been shown in [9] that any 
solution is a local minimizer of ([3]) as long as it satisfies the first and second order necessary 
optimality conditions of However, Example 2.1 shows that this fact is not true for L2-Lp 
problem ([TJ. 

Thus, we need somewhat new proofs for the hardness result. To facilitate the new proof, we 
first prove that problem ([5]) is NP-hard, and then extend to the strongly NP-hard result. 

Theorem 3. Minimization problem ^ is NP-hard for any given < p < 1, (7>1 and A > 0. 

We first prove a useful technical lemma. 

Lemma 4. Consider the problem 

Minimizez^R g{z) := |1 — zj"^ + 21-^1^ (^^) 

for some given < p < 1 and q > 1. It is minimized at a unique point (denoted by z*{p, q) ) on 
(0, 1]. And the optimal value c{p,q) is less than |. 

Proof. First it is easy to see that when p = 0, g{z) has a unique minimizer at z = 1, and the 
optimal value is ^. Now we consider the case when p 7^ 0. Note that g{z) > g{0) = 1 for all 
z < 0, and g{z) > g{l) = ^ for all z > 1. Therefore the minimum point must lie within [0, 1]. 
To optimize g{z) on [0, 1], we check its first derivative 

g'{z) = -q[l-z)'>-^ + P^. (15) 

We have ^'(O"'') = +00 and g'{l) = | > 0. Therefore, if function g{z) has at most two stationary 
points in (0,1), the first one must be a local maximum and the second one must be the unique 
global minimum and the minimum value c{p,q) must be less than ^. 
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Now we check the possible stationary points of g{z). Consider solving g'{z) = —q{l — z)'^ ^ + 
2^ = 0. We get z^-P{l - = ^. 

Define h{z) = z^-p{1 - We have 

h'{z) = h{z){^P-'j^J. 

Note that — is decreasing in z and must have a root on (0, 1). Therefore, there exists 
a point z G (0, 1) such that h'(z) > for z < z and h'{z) < for z > z. This implies that 
h{z) = ^ can have at most two solutions in (0,1), i.e., g{z) can have at most two stationary 
points. By the previous discussions, the lemma holds. □ 

Proof of Theorem [3l First we claim that without loss of generality we only need to consider 
the problem with A = ^. This is because given any problem of form ([5]), we can make the 
following transformation: 

X = {2\)^/Px , A = {2X)-^/PA and 6 = 6 

and scale this problem to: 

Minimize^ - + (16) 

Note that this transformation is invertible, i.e., for any given Aq, one can transform an instance 
with A = Aq to one with A = ^ and vice versa. Therefore, we only need to consider the case 
when A = ^. 

Now we present a polynomial time reduction from the well known NP-complete partition 
problem [8\ to problem ()16p . The partition problem can be described as follows: given a set S 
of rational numbers {ai,a2, ■ ■ ■ ,&„}, is there a way to partition S into two disjoint subsets Si 
and 5*2 such that the sum of the numbers in Si equals to the sum of the numbers in 

Given an instance of the partition problem with a = (01,02, . . . ,anY' ^ We consider 
the following minimization problem in form (|16p : 

Minimize^,2, P{x,y) = \ J {x - y)\i + ^ \xj + yj-l\'' + - ^ {\xj\P + \yj\P). (^7) 
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We have 



Minimizex.,yP(x, y) > Minimize^^.^ ^ |xj + - 1|^ + ^ ^ {\xj\P + \yj\P) 

= ^ Minimizea:^.,^^. \xj + yj - 1|^ + ]^{\xj\P + \yj\P) 
l<j<n 

= n ■ Minimize^ |1 — zl"^ H — \z\P , 



where the last equality is from the fact that \xj\P + \yj\P > \xj + yj\P and that we can always 
choose one of them to be such that the equality holds. 
By applying Lemma IU we have 

P{x,y) > nc{p,q). 
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Now we claim that there exists an equitable partition to the partition problem if and only if the 
optimal value of (fTUj) equals to nc{p,q). First, if S can be evenly partitioned into two sets Si 
and 52, then we define (xj = z*{p, q),yi = 0) if Oj belongs to Si and define (xj = 0,yi = z*{p, q)) 
otherwise. These {xj,yj) provide an optimal solution to P{x,y) with optimal value nc{p,q). 
On the other hand, if the optimal value of ([5]) is nc{p,q), then in the optimal solution, for each 
i, we must have either (xj = z*(j),q),yi = 0) or (xj = 0,yi = z*{p,q)). And we must also have 
a'^{x — y) = 0, which implies that there exists an equitable partition to set S. Thus Theorem 
[3] is proved. □ 

In the following, using the similar idea, we prove a stronger result: 

Theorem 5. Minimization problem ^ is strongly NP-hard for any given 0<p<l, q>l 
and A > 0. 

Proof. We present a polynomial time reduction from the well known strongly NP-hard 3- 
partition problem [318]. The 3-partition problem can be described as follows: given a multiset 
S of n = 3m integers {oi, 02, ... , On} with sum mB, can S be partitioned into m subsets, such 
that the sum of the numbers in each subset is equal? 

We consider the following minimization problem in the form (|16p : 

m n n m ^ n m 

Minimize P{x) = Y,\Y1 " ^1' + 2^ I " ^l" + 2 2^ l^^il"- ^1^) 

j = l i=l i=l j = l i=l j = l 

The remaining argument will be the same as the proof for Theorem [3l □ 

Theorem [5] implies that the L2-Lp minimization problem is strongly NP-hard. Next we 
generalize the NP-hardness result to the smoothed version of this problem in ([6]) . 

Theorem 6. Minimization problem (0j is strongly NP-hard for any give < p < 1, q > 1, 
A > and e > 0. 

Proof. We again consider the same 3-partition problem, we claim that it can be reduced to a 
minimization problem in form ([B]). Again, it suffices to only consider the case when A = ^ (Here 
we consider the hardness result for any given e > 0. Note that after the scaling, e may have 
changed). Consider: 



m n n m ^ n m 

Minimize, P,(x) = ^ | ^ a^Xi,- - + ^ | J]] x,, - + - J]] | + . (19) 

j=l i=l i=l j=l i=l j=l 

We have 

n m ^ n m 

Minimizea;-Pe(x) > Minimize^; | Xjj — 

i=l j=l 1=1 j=l 

n m ^ m 

= ^ Minimize, | " ^1' + 2 IZd^^il + 

i=l j=l j=l 

I (m — 1) 

= n ■ Minimize^ \l - z\'^ + -{\z\ + ef + -^^ — - — -e^. 

The last equality comes from the submodularity of the function (x+e)^ and the fact that one can 
always choose only one of Xij to be nonzero in each set such that the equality holds. Consider 
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function ge{z) = \l — z]"^ + ^{\z\ +e)^. Similar to Lemmadl one can prove that ge{z) has a unique 
minimizer in [0,1]. Denote this minimum value by c{p,q,e), we know that -Pe(x) > nc{p,q,€). 
Then we can argue that the 3-partition problem has a solution if and only if Pe{x) = nc{p, q, e). 
Therefore Theorem O holds. □ 

The above results reveal that finding a global minimizer for the Lq-Lp minimization problem 
is strongly NP-hard, or the original sparse least squares problem is intrinsically hard, and no 
regularized optimization models/methods could help much in the worst case. That is, relaxing 
Lq to Lp for some < p < 1 in the regularization gains no significant advantage in terms of the 
(worst-case) computational complexity. 



4 Bounds (5{k) and ^{k) for asymptotic properties 

Given the strong negative result for computing a global minimizer, our hope now is to find a 
local minimizer of problem ([1]), still good enough for the desired sparsity - say no more than 
k nonzero entries. This is indeed guaranteed by Theorem [2] if one chooses A > ^{k) of (jlip . 
instead of A > P{k) of ([7]). In the following, we present a positive result in the bridge estimator 
model considered by [H [ini IH] . 

Consider asymptotic properties of the L2-Lp minimization ([1]) where the sample size m tends 
to infinity in the model of [H lllj . Suppose that the true estimator x* has no more than k 
nonzero entries. One expects that there is a sequence of bridge estimators, i.e. solutions xj^ of 

Minimize \\Ax - + A^lkH^ 

such that dist(support{a;J^}, supportjx*}) — 0, as m — )■ c«, with probability 1. 

In applications of variable selection, the design matrix is typically standardized so that 

||aj||^ = m for i = l,...,n. 

Moreover, the smallest and largest eigenvalues pi and p2 of the covariate matrix = ^A^A 
satisfy < ci < pi < p2 < C2 for some constants ci and C2, see |10]. This assumption implies 
that ^c\m < \\A\\ < yjcqrri. For simplicity, let us fix ||yl|| = yfm and p = 1/2. Then we have 

I3{k) = A;-3/4(8m)i/4||6f /2 and -f{k) = k-^/\l6m)^/^\\bf/^. 

One can see that j{k) > f3{k) for all k > 1. 

If /c is a constant, we see that /3{k) and j{k) are in the same order of m and ||5||. Thus, 
finding any local minimizer of problem ([TJ in the objective level set /p(0) is sufficient to guarantee 
desired sparsity when Am = /3{k). That is, there is no significant guaranteed sparsity difference 
between global and local minimizers of problem ([T]). This seems also observed in computational 
experiments when the true estimator is extremely sparse. Of course, when k increases as 
m —7- oo, a global minimizer of problem ([T]) would likely become sparser than its local minimizer, 
since P{k)/j{k) = 0{k-^/^). 

In general, both /3{k) and 7(/c) meet the conditions in the analysis of consistency and oracle 
efficiency of bridge estimators of [lOl lllj. In their model, the parameter Am is required to satisfy 
certain conditions. For instances, 

{^Theorems]) \mm~^l'^ ^ Aq > as m ^ oo (20) 
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i^A3,{a)]) Amm"^/2 ^ as m ^ oo. (21) 

With ||aj|p = m for i = 1, . . . ,n and ||^|| = ^/m in their model, we have 

p/2 



\p{l 



P) 

and 



P ^ An > as m oo 



/ o \ p/2 

/3(A;)m-i/2 ^ ^p/2-i ^ ^ ,,^,|2_p^(p_i)/2 ^ as m ^ oo. 



For j{k), we have 

-1/^1 l|/,||2-p 



Ao > as m — )• oo 



and 



|2-P^(p-i)/2 ^ as m^oo. 



Hence, both Am = (3{k) and Am = "f{k) satisfy (PU|) and (pij) . Moreover, by Theorem [T] 
and Theorem [2l any minimizer of L2-Lp problem ([TJ with A = Am is likely to have less than k 
nonzero entries. Hence each of them could be a good choice for consistency and oracle efficiency 
of bridge estimators via solving the unconstrained L2-Lp minimization problem ([1]) . 
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